Method and system for training a user how to perform gestures

ABSTRACT

A system and method for determining whether a flick gesture has occurred is described. A flick gesture is a simple gesture that may be easily detected and is characterized by minimal interference with other applications or gestures.

RELATED APPLICATION INFORMATION

This application is a continuation of U.S. Ser. No. 10/938,882, filedSep. 13, 2004, to Emily Rimas-Ribikauskas, Rob Jarrett, Jeff Pettiross,Matthew Lerner, Leroy B. Keely, Rick Duncan, and Sriram Viji, whosecontents are expressly incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Aspects of the present invention relate to computing systems. Moreparticularly, aspects of the present invention relate to a process fordetermining when a hand-written gesture has occurred.

2. Description of Related Art

Computing systems accept a variety of inputs. Some computer applicationsaccept gestures formed by pointing devices to enable easier control andnavigation of the applications. The difficulty however with conventionalgesture recognizers is that they provide too many false positives andfalse negatives. When a user is using a pointing device (for instance, astylus or finger with a stylus-based computer or a mouse), a system mayinterpret normal actions from the pointing device as gestures andimplement the functions associated with the gestures to the bewildermentof the user. Likewise, a user may become frustrated when trying toperform a gesture and the computing system not recognizing the user'seffort as a gesture. In some cases, gestures are relegated to a separateoperation where a user is forced to change a mode of a system before agesture may be recognized. For instance, the Opera browser (by OperaSoftware ASA of Oslo, Norway) permits mouse gestures to be used tocontrol the browser's operation. Similar mouse gestures are availablethrough plug-ins with the Firefox browser (by the Mozilla Organization,Mountain View, Calif.). While these browsers recognize gestures, theuser is required to switch operating modes of the browsers in order toexplicitly notify the browsers that gestures are to be performed. Inboth the Opera and Firefox browsers, the operating mode is switched bythe user clicking on a right (or auxiliary) mouse button prior toperforming the gesture. The issue here is that multiple steps have to beperformed in order to complete all gestures, namely press a right mousebutton and then drag the mouse. When using a stylus, these browsersrequire the stylus to enter an alternative input mode, namely pressing astylus button (if the stylus has a stylus button). These multiple stepsmake gestures cumbersome. Further, while these browsers may recognize agesture composed of a straight line while depressing a mouse or stylusbutton, gestures acceptable on browsers are generally not usable in moreenvironments because they would generate too many false positives whenapplied to non-read only environments (for instance, a word processingenvironment).

In computer systems or computer applications that recognize and assignfunctions to gestures, shape alone is generally used to determinewhether or not a gesture has occurred. An issue with using shape aloneis that slow stylus or mouse-wielding users may have a tendency toperform actions that may be recognized as gestures, despite theintentions of the users to the contrary.

Marking menus (also referred to as radial menus) are menus that may beaccessed through interaction with a pointing device. Maya and Aliasapplications of Alias Systems Corp. of Toronto, Ontario, Canada provideexamples of marking menus. While similar to gestures in that theyinclude start and stop locations, marking menus are considered menus inthat the relevant part of the user input is the location of a button upor stylus up event, not the shape of the actual path to the location.

An improved gesture is needed that is easily recognized by a gesturerecognizer yet easy to perform by a user.

BRIEF SUMMARY OF THE INVENTION

Aspects of the present invention address one or more of the problemsdescribed above, thereby providing a process for robustly recognizinggestures.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and not limitedin the accompanying figures in which like reference numerals indicatesimilar elements and in which:

FIG. 1 shows an illustrative example of a general-purpose computingenvironment in accordance with aspects of the present invention.

FIG. 2 shows an illustrative example of a tablet computer in accordancewith aspects of the present invention.

FIG. 3 shows various regions in which a flick gesture may be determinedto have occurred in accordance with aspects of the present invention.

FIG. 4 shows processes for determining whether a flick gesture hasoccurred in accordance with aspects of the present invention.

FIG. 5 shows an illustrative tutorial process for providing instructionto a user in accordance with aspects of the present invention.

FIG. 6 shows various illustrative components that may be used with agesture recognizer in accordance with aspects of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Aspects of the present invention relate to recognizing a flick gesture.

This document is divided into sections to assist the reader. Thesesections include: characteristics of ink, terms, general-purposecomputing environment, gestures, a flick gesture, flick gesturedetermination, training, and illustrative implementations.

It is noted that various connections are set forth between elements inthe following description. It is noted that these connections in generaland, unless specified otherwise, may be direct or indirect and that thisspecification is not intended to be limiting in this respect.

Characteristics of Ink

As known to users who use ink pens, physical ink (the kind laid down onpaper using a pen with an ink reservoir) may convey more informationthan a series of coordinates connected by line segments. For example,physical ink can reflect pen pressure (by the thickness of the ink), penangle (by the shape of the line or curve segments and the behavior ofthe ink around discreet points), and the speed of the nib of the pen (bythe straightness, line width, and line width changes over the course ofa line or curve). Further examples include the way ink is absorbed intothe fibers of paper or other surface it is deposited on. These subtlecharacteristics also aid in conveying the above listed properties.Because of these additional properties, emotion, personality, emphasisand so forth can be more instantaneously conveyed than with uniform linewidth between points.

Electronic ink (or ink) relates to the capture and display of electronicinformation captured when a user uses a stylus-based input device.Electronic ink refers to a sequence or any arbitrary collection ofstrokes, where each stroke is comprised of a sequence of points. Thestrokes may have been drawn or collected at the same time or may havebeen drawn or collected at independent times and locations and forindependent reasons. The points may be represented using a variety ofknown techniques including Cartesian coordinates (X, Y), polarcoordinates (r, Θ), and other techniques as known in the art. Electronicink may include representations of properties of real ink includingpressure, angle, speed, color, stylus size, and ink opacity. Electronicink may further include other properties including the order of how inkwas deposited on a page (a raster pattern of left to right then down formost western languages), a timestamp (indicating when the ink wasdeposited), indication of the author of the ink, and the originatingdevice (at least one of an identification of a machine upon which theink was drawn or an identification of the pen used to deposit the ink)among other information.

Terms

-   -   Ink A sequence or set of strokes with properties. A sequence of        strokes may include strokes in an ordered form. The sequence may        be ordered by the time captured or by where the strokes appear        on a page or in collaborative situations by the author of the        ink. Other orders are possible. A set of strokes may include        sequences of strokes or unordered strokes or any combination        thereof. Further, some properties may be unique to each stroke        or point in the stroke (for example, pressure, speed, angle, and        the like). These properties may be stored at the stroke or point        level, and not at the ink level    -   Ink object A data structure storing ink with or without        properties.    -   Stroke A sequence or set of captured points. For example, when        rendered, the sequence of points may be connected with lines.        Alternatively, the stroke may be represented as a point and a        vector in the direction of the next point. In short, a stroke is        intended to encompass any representation of points or segments        relating to ink, irrespective of the underlying representation        of points and/or what connects the points.    -   Point Information defining a location in space. For example, the        points may be defined relative to a capturing space (for        example, points on a digitizer), a virtual ink space (the        coordinates in a space into which captured ink is placed),        and/or display space (the points or pixels of a display device).    -   Document Any electronic file that has a viewable representation        and content. A document may include a web page, a word        processing document, a note page or pad, a spreadsheet, a visual        presentation, a database record, image files, and combinations        thereof.    -   Flick A short, quick, straight movement of a pointing device of        a computer system. Where the pointing device is a stylus or        finger, the flick may occur on the surface of the stylus-enabled        computer or may occur above its surface. Where the pointing        device is a mouse, the movement may occur with a mouse button        being held down or without a mouse button being held down. The        flick may create a stroke as defined above. It is noted that        strokes may be created by other activities as well including but        not limited to drawing or inking with a pointing device. An        advantage of using a stylus to perform a flick gesture is that        it permits one to use flick gestures in combination with        pen-enabled computers (using active pens and static pens). An        advantage of using a mouse to perform a flick gesture is that a        mouse is easy to use and commonly found with most computers. An        advantage of using a finger to perform a flick gesture is that        some users prefer using a finger as compared to a stylus on        pen-based computers —allowing users to use their finger to        perform flicks provides them the functionality of a flick        without using additional hardware.        General-Purpose Computing Environment

FIG. 1 illustrates an example of a suitable computing system environment100 on which the invention may be implemented. The computing systemenvironment 100 is only one example of a suitable computing environmentand is not intended to suggest any limitation as to the scope of use orfunctionality of the invention. Neither should the computing environment100 be interpreted as having any dependency or requirement relating toany one or combination of components illustrated in the exemplaryoperating environment 100.

The invention is operational with numerous other general purpose orspecial purpose computing system environments or configurations.Examples of well known computing systems, environments, and/orconfigurations that may be suitable for use with the invention include,but are not limited to, personal computers, server computers, hand-heldor laptop devices, multiprocessor systems, microprocessor-based systems,set top boxes, programmable consumer electronics, network PCs,minicomputers, mainframe computers, distributed computing environmentsthat include any of the above systems or devices, and the like.

The invention may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc., that performparticular tasks or implement particular abstract data types. Theinvention may also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices.

With reference to FIG. 1, an exemplary system for implementing theinvention includes a general purpose computing device in the form of acomputer 110. Components of computer 110 may include, but are notlimited to, a processing unit 120, a system memory 130, and a system bus121 that couples various system components including the system memoryto the processing unit 120. The system bus 121 may be any of severaltypes of bus structures including a memory bus or memory controller, aperipheral bus, and a local bus using any of a variety of busarchitectures. By way of example, and not limitation, such architecturesinclude Industry Standard Architecture (ISA) bus, Micro ChannelArchitecture (MCA) bus, Enhanced ISA (EISA) bus, Video ElectronicsStandards Association (VESA) local bus, and Peripheral ComponentInterconnect (PCI) bus also known as Mezzanine bus.

Computer 110 typically includes a variety of computer readable media.Computer readable media can be any available media that can be accessedby computer 110 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes both volatileand nonvolatile, and removable and non-removable media implemented inany method or technology for storage of information such as computerreadable instructions, data structures, program modules or other data.Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can accessed by computer 110. Communication media typicallyembodies computer readable instructions, data structures, programmodules or other data in a modulated data signal such as a carrier waveor other transport mechanism and includes any information deliverymedia. The term “modulated data signal” means a signal that has one ormore of its characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media. Combinations of the any of the aboveshould also be included within the scope of computer readable media.

The system memory 130 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 131and random access memory (RAM) 132. A basic input/output system 133(BIOS), containing the basic routines that help to transfer informationbetween elements within computer 110, such as during start-up, istypically stored in ROM 131. RAM 132 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 120. By way of example, and notlimitation, FIG. 1 illustrates operating system 134, applicationprograms 135, other program modules 136, and program data 137.

The computer 110 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 1 illustrates a hard disk drive 141 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 151that reads from or writes to a removable, nonvolatile magnetic disk 152,and an optical disk drive 155 that reads from or writes to a removable,nonvolatile optical disk 156 such as a CD ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks, digital video tape, solid state RAM, solidstate ROM, and the like. The hard disk drive 141 is typically connectedto the system bus 121 through a non-removable memory interface such asinterface 140, and magnetic disk drive 151 and optical disk drive 155are typically connected to the system bus 121 by a removable memoryinterface, such as interface 150.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 1, provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 110. In FIG. 1, for example, hard disk drive 141 is illustratedas storing operating system 144, application programs 145, other programmodules 146, and program data 147. Note that these components can eitherbe the same as or different from operating system 134, applicationprograms 135, other program modules 136, and program data 137. Operatingsystem 144, application programs 145, other program modules 146, andprogram data 147 are given different numbers here to illustrate that, ata minimum, they are different copies. A user may enter commands andinformation into the computer 20 through input devices such as akeyboard 162 and pointing device 161, commonly referred to as a mouse,trackball or touch pad. Other input devices (not shown) may include amicrophone, joystick, game pad, satellite dish, scanner, or the like.These and other input devices are often connected to the processing unit120 through a user input interface 160 that is coupled to the systembus, but may be connected by other interface and bus structures, such asa parallel port, game port or a universal serial bus (USB). A monitor191 or other type of display device is also connected to the system bus121 via an interface, such as a video interface 190. In addition to themonitor, computers may also include other peripheral output devices suchas speakers 197 and printer 196, which may be connected through anoutput peripheral interface 195.

The computer 110 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer180. The remote computer 180 may be a personal computer, a server, arouter, a network PC, a peer device or other common network node, andtypically includes many or all of the elements described above relativeto the computer 110, although only a memory storage device 181 has beenillustrated in FIG. 1. The logical connections depicted in FIG. 1include a local area network (LAN) 171 and a wide area network (WAN)173, but may also include other networks. Such networking environmentsare commonplace in offices, enterprise-wide computer networks, intranetsand the Internet.

When used in a LAN networking environment, the computer 110 is connectedto the LAN 171 through a network interface or adapter 170. When used ina WAN networking environment, the computer 110 typically includes amodem 172 or other means for establishing communications over the WAN173, such as the Internet. The modem 172, which may be internal orexternal, may be connected to the system bus 121 via the user inputinterface 160, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 110, orportions thereof, may be stored in the remote memory storage device. Byway of example, and not limitation, FIG. 1 illustrates remoteapplication programs 185 as residing on memory device 181. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

In some aspects, a pen digitizer 165 and accompanying pen or stylus 166are provided in order to digitally capture freehand input. Although adirect connection between the pen digitizer 165 and the user inputinterface 160 is shown, in practice, the pen digitizer 165 may becoupled to the processing unit 110 directly, parallel port or otherinterface and the system bus 130 by any technique including wirelessly.Also, the pen 166 may have a camera associated with it and a transceiverfor wirelessly transmitting image information captured by the camera toan interface interacting with bus 130. Further, the pen may have othersensing systems in addition to or in place of the camera for determiningstrokes of electronic ink including accelerometers, magnetometers, andgyroscopes.

It will be appreciated that the network connections shown areillustrative and other techniques for establishing a communications linkbetween the computers can be used. The existence of any of variouswell-known protocols such as TCP/IP, Ethernet, FTP, HTTP and the like ispresumed, and the system can be operated in a client-serverconfiguration to permit a user to retrieve web pages from a web-basedserver. Any of various conventional web browsers can be used to displayand manipulate data on web pages.

FIG. 2 illustrates an illustrative tablet PC 201 that can be used inaccordance with various aspects of the present invention. Any or all ofthe features, subsystems, and functions in the system of FIG. 1 can beincluded in the computer of FIG. 2.

Tablet PC 201 includes a large display surface 202, e.g., a digitizingflat panel display, preferably, a liquid crystal display (LCD) screen,on which a plurality of windows 203 is displayed. Using stylus 204, auser can select, highlight, and/or write on the digitizing displaysurface 202. Examples of suitable digitizing display surfaces 202include electromagnetic pen digitizers, such as Mutoh or Wacom pendigitizers. Other types of pen digitizers, e.g., optical digitizers, mayalso be used. Tablet PC 201 interprets gestures made using stylus 204 inorder to manipulate data, enter text, create drawings, and/or executeconventional computer application tasks such as spreadsheets, wordprocessing programs, and the like.

The stylus 204 may be equipped with one or more buttons or otherfeatures to augment its selection capabilities. In one embodiment, thestylus 204 could be implemented as a “pencil” or “pen”, in which one endconstitutes a writing portion and the other end constitutes an “eraser”end, and which, when moved across the display, indicates portions of thedisplay are to be erased. Other types of input devices, such as a mouse,trackball, or the like could be used. Additionally, a user's own fingercould be the stylus 204 and used for selecting or indicating portions ofthe displayed image on a touch-sensitive or proximity-sensitive display.Consequently, the term “user input device”, as used herein, is intendedto have a broad definition and encompasses many variations on well-knowninput devices such as stylus 204. Region 205 shows a feedback region orcontact region permitting the user to determine where the stylus 204 ascontacted the display surface 202.

In various embodiments, the system provides an ink platform as a set ofCOM (component object model) services that an application can use tocapture, manipulate, and store ink. One service enables an applicationto read and write ink using the disclosed representations of ink. Theink platform may also include a mark-up language including a languagelike the extensible markup language (XML). Further, the system may useDCOM as another implementation. Yet further implementations may be usedincluding the Win32 programming model and the .Net programming modelfrom Microsoft Corporation.

Gestures

Gestures are ways to invoke an action, similar to clicking a toolbarbutton or typing a keyboard shortcut. Gestures may be performed with apointing device (including but not limited to a mouse, stylus, and/orfinger). A gesture has a shape associated with it. The shape may be assimple as a point (clicking a mouse button for instance) or ascomplicated as a series of movements. The gesture also has a positionassociated with it as well. The position may be used to dictate thescope, target, and/or meaning of the gesture.

A Flick Gesture

Gestures have been used in computing systems for years. However, manydevelopers and end-users disable gestures because of difficulties andcomplexities created by gestures. For instance, gesture recognitionengines suffer from an unacceptably high number of false positives andfalse negatives when recognizing gestures. These false positives (wherea system determines a user input is a gesture when the user input wasnot intended as a gesture) and false negatives (where a systemdetermines a user input not to be a gesture when a user has tried toperform a gesture) frustrate developers and end-users alike.

A flick gesture, as described herein, is a simple gesture that includesa single movement of a pointing device. A flick gesture is easy for theuser to remember and perform. Once a user has mastered a flick gesture,it can be applied in multiple directions to accomplish different tasks.Also, a flick gesture is easy for the system to recognize anddiscriminate against other actions (including drag, selection, etc.)

A developer and/or user may associate operations with the flick gesture.These operations may include navigation forward, backward, scrolling upor down, changing applications, right click (which may or may not alwaysbe present in a stylus-based system), and arbitrary applicationcommands. Arbitrary application commands are commands pertinent toapplications that are specific to those applications and not generallyusable across other applications. Further, a flick gesture does not needto have a predefined meaning but rather may be customizable by adeveloper or user to perform a key or key combination so that a user mayhave quick access to keyboard shortcuts or macros. One benefit ofenabling gestures is that it allows users to feel more productive, toovercome feelings of being unproductive when not using a keyboard.

The flick gesture may be consistent in its associated function acrossall applications in an operating system. Alternatively, a flick gesturemay be contextual in the function associated with it (where theresulting operation tied to the flick gesture varies based on anapplication in which the flick gesture occurred).

Also, flick gestures may be specific to locations in an application (forinstance, in a title bar) or whether a selection has been made or not(for instance, the gesture may perform cut, copy, paste, and/or pastespecial operations). Additionally or alternatively, states of a computermay be used in determining which action associated with a flick gestureshould be performed. For instance, a color of a currently selectedobject, which user is logged into the system, whether an externalmonitor is connected to the system, and the like—in short, any state orcombination of states of a computing system or object—may be used todetermine what action is to be performed in response to a flick gesture.

Further, different input devices may modify actions associated withflick gestures. For instance, a first set of actions may be associatedwith flick gestures when performed by a stylus. A second set of actionsmay be associated with flick gestures when performed by another pointingdevice. The number of sets of actions may be varied by the number ofdifferent input devices.

Further, hot zones may or may not be enabled where a flick in the hotzone may have yet another operation. Hot zones may include specificregions of a display and/or radial menus.

The flick gesture may be direction independent or may be directionspecific. If direction specific, the direction the flick is drawn inwill determine the outcome. The flick gesture characteristics may or maynot vary based on direction. For instance, the system may have only oneset of parameters for recognizing a flick gesture. Alternatively, thesystem may have any number of parameters that vary depending on thedirection of the flick to accommodate left or right-handedness, angle ofa stylus, and the like. These parameters may be modified to bias theease of performing a flick gesture in certain directions.

Once the user learns how to perform flick gesture, she is only requiredto learn the actions linked to each of the available directions. Theactions associated with each of the directions may be customizableallowing a user to have a personalized and optimized experience.

Flick gestures may be used alone. Alternatively, the flick gestures maybe combined to perform additional actions. For instance, a flick gestureupwards may perform a page up action while two flick gestures both inthe upwards direction may perform a move action that moves to the top ofa document.

In alternative embodiments, flick gestures may be enabled throughpressing a button on a stylus or by pressing a right mouse button(secondary mouse button) on a mouse while still satisfying any movementand time thresholds. However, one advantage of the flick gesture is thatit may be performed with a left mouse button click or with a stylus orfinger without changing modes. The flick gesture may increase a user'sproductivity and allows them to interact with their desktop andapplications at a more effective level than was previously available.The gestures are designed such that a user has to do a minimal level oftraining/remembering in order to gain a significant benefit; once thegesture is mastered by a user they are only required to learn thedirection-action association. Distinguishing between normal mouse orstylus movements and a flick gesture may be made based on a variouscriteria and constraints placed on the flick gesture.

Flick gestures may allow users to dispense with targeting scrollbars ornavigation buttons. This may include reading documents or navigatingdocuments with or without hyperlinks. Reading documents of this type maybe performed online and offline. Additionally, common actions includingcut, copy, and paste may be performed using flick gestures.

Flick Gesture Determination

A flick gesture can be performed by a user simply by flicking their penagainst the screen; any active portion of the pen could be used,depending on the implementation (tip or bottom). Flick gestures may beperformed in the natural mode without necessarily requiring the user toenter any modes—although a mode requirement may be made, for example,requiring the user to hold the pen button while performing a flickgesture. Flick gestures can be done in 8 directions (for instance), witheach flick falls into one of the 8 directions. The occurrence of a flickgesture may be determined based on a profile of the physical or logicalx and y co-ordinates and the pressure (or location) charted againsttime. In short, the three basic characteristics that may be used todetermine whether a flick gesture has occurred include the direction ofthe flick, the movement of the flick, and the timing of the flick. Themovement threshold may be, for example, greater than 1 cm and the timethreshold greater than 0.2 ms. These values of course may be varied toaccommodate all users. Using these characteristics, a system maycalculate other aspects of a flick which may be more readily comparedagainst thresholds. For instance, the system may calculate velocity,acceleration, curvature, lift, and the like and use these derived valuesor sets of values to determine if a user has performed a flick gesture.

Flick gestures may be determined apart from other actions including adouble actuation (double tap or click), press and hold, hold and drag,and other gestures.

Other methods such as neural networks may also be used to attempt toclassify flick gestures based on combinations of input characteristicsderived from the three basic characteristics.

One of the benefits of using these criteria to determine whether a flickgesture has occurred is the minimization of false negatives and falsepositives while minimizing the detection time of the flick gestures. Incombination, these benefits provide for overall improved systemresponsiveness. For instance, the procedure for determining whether aflick has occurred may be organized so as to quickly determine that aflick does not meet the qualifications for a flick gesture and permitthe input device movement to be treated as a normal input devicemovement.

Aspects of the present invention may include rendering a flick movementand/or action associated with a normal input device action while thesystem is determining whether or not the flick is indeed a flickgesture. Alternatively, aspects of the present invention may includepreventing input information (received from the user operating thepointing device) from being processed or rendered until the systemdetermines whether or not a flick gesture has occurred. While thislatter approach may cause a slight, visual delay in feedback to a user,it may improve the user's ability to perform flick gestures in thepresence of objects that may be manipulated with a normal input. Forinstance, performing a flick over an object may be processed as a normalinput, resulting in dragging and dropping the object, while a flickgesture may have been intended. Accordingly, these alternative aspectsof the invention may hold the packets (that may or may not be a flickgesture) until the system determines whether they are a flick gesture ornot.

FIG. 3 shows the various illustrative directions that may be used forthe flick gesture. The group of all directions is shown by region 301.Four Cardinal regions 302-305 are shown as well as four diagonal regionsare shown 306-309. The system may treat all eight regions identical inrespect to determination of which direction a flick gesture hastraveled. Alternatively, some regions (for instance, the Cardinaldirections) may be biased to classify a users input as falling into oneof these directions over the diagonal directions. The number ofdifferent directions may be fixed or modifiable. For instance, one mayalternatively use 6 or 12 or any number of directions as the number ofdirections available to a user.

In order to determine the flick direction, the pen down point representthe center of a circle, there may be 8 equally shaped and spaced regionssurrounding the center, each a 45 degree section of the circle, forinstance. These regions are not dependent on the orientation of thetablet, and rotate accordingly when tablet is rotated.

The regions do not need to be restricted to 45 degrees, and could belarger or smaller depending on the application. However, the ones usedby default throughout the system may initially be set at 45 degrees.

A flick gesture, by definition, starts at the center of the circle shownin FIG. 3, as the pen down (or left mouse click) point is the center;and a short, quick, straight stroke, termed a flick, is drawn out forthe pen down point. By definition of a flick, being short and straight,it will fall into one of the regions. Whichever region it is drawn inmay determine the action which is fired. A flick region is determined byconnecting the starting and ending points of the stroke which are incontact with the screen, provided the stroke meets a quality of a flick(which may or may not include the straightness, length and speedrequirement as set forth herein). If the flick falls on the exact linewhich separates the regions, the flick may default to the regionsnon-diagonal region, or those representing the horizontal or verticaldirection.

Upon successful recognition of a flick gesture, feedback may or may notbe provided to a user, where the feedback indicates that the flickgesture a user has just performed has been recognized and that a messageis being sent to the operating system or application. The feedback mayalso indicate the action that has been fired. This feedback does notmean that an action will take place, only that a successful flick hasbeen recognized and is being sent. If, for example, a user continues toperform page down flick gestures once she has reached the bottom of thepage, she will still get the correct feedback although no action will beresulting as they can scroll no further down the page.

This feedback may appear as soon as a successful gesture is recognizedand will remain for a short period of time (in the vicinity of 500 ms),as which time it will disappear. Alternatively, no feedback may beprovided. Further, in additional alternate aspects, the feedback may beanimated to more readily inform a user that a successful flick gesturehas been recognized. An animation may or may not provide the user withfeedback of a direction associated with the recognized flick gesture.

FIG. 4 shows a procedure for determining if a flick has occurred. Instep 401, a pen down or button down event occurs. Next in step 404, thesystem determines if there was movement beyond a movement threshold (forinstance, beyond 0.2-0.5 cm). If no, then the system may determine thatthe input being received is some other action or selection as shown instep 405. If there was movement beyond the threshold from step 404, thenthe system determines if a velocity of the input device has droppedbelow a velocity threshold (for instance 20% or less of the highestvelocity or 5 cm per second or less). The drop may or may not besensitive for a drop in the velocity for brief interval (10-200 ms).Alternatively, the process may determine if the velocity of the receivedgesture is not decreasing.

If no from step 406, then another action or selection is determined instep 407. If yes from step 406, then a flick gesture is determined tohave occurred in step 408. The current location may be considered to beany location along the path of the flick including but not limited tothe location of the stylus or finger being lifted from a screen or mousebutton being released.

The following describes optional additions to the process of FIG. 4.From step 401, the system may determine if a pen down location was in aninking region. If yes, then the system may believe the pen down event torelate to the creation of ink as shown in step 403. This may or may notinclude additional operations (for example actually creating ink,selecting, or erasing ink).

Another optional step includes step 409. In step 409, the systemdetermines if the path of the possible flick gesture is relativelystraight. This may include a determination of whether all points in thegesture lie within a predefined angle from an initial contact point.This initial angle may include 45° or less. If no, then the process mayconclude that the input is not a flick gesture in step 410.

A further optional set of steps (411 and 412) includes analyzing themotion of the pointing device before and/or after it contacts thescreen. For example, by noticing that a stylus continues in a straightline for some distance after the stylus tip or leaves the screen, onemay further refine the precision of the detection algorithm. Theapproach of steps 409 and 412 help to quickly get rid of flicks that arenot flick gestures. Alternatively, the approach of steps 409 and 411,while more complex and functionally slower (because this alternativeapproach may interpret flicks as flick gestures that had been rejectedbefore), may be more tolerant of user movements.

Once a successful flick has taken place, the system may alert otheraspects of the system or application that a flick gesture has occurred.A message may be sent such that an application can perform theassociated behavior with that action, if there is no assigned behaviorthe flick architecture may send a default value for the application tointerpret. For example, if a Back command is sent to an application,this may be interpreted as Back command in the browser, or as previousentry in a database application. More particularly, a variety ofdifferent approaches may be applied to handling flick gestures. First, aflick gesture may be interpreted by the operating system and handledthere. The mapping from a given direction to the semantic may beperformed by the system or may be omitted. Second, the mapping of theflick to a flick action may be made in the operating system but sent tothe application or applications for handling. The system may communicatejust the original direction of the flick (either by indicating apredefined region or the exact detected angle of the flick) to one ormore applications programs, it may communicate the semantic, orcommunicate both. Third, the system may communicate the flick to anapplication or application where the application or applicationsinterpret the flick as a flick gesture. The application program orprograms may interpret the predefined region, exact angle, semanticmessage, or some combination of the three. The application orapplications can then handle the flick gesture or send it back to theoperating system for handling.

Flick gestures may apply at the pen down location of the flick or to thecurrent focus, foreground application, or cursor location. Flickgestures are not limited to navigation but may be used to perform otheroperations as well. For instance, user may want to have a flick gestureperform a right-click operation at the location of a stylus contacting ascreen.

Developers may be able to turn flick gestures off in their applicationsor in a specific area of their applications, such as scrollbars and thelike. In addition, developers may be prevented from customizing flickgestures or, alternatively, may be able to customize flick gestures totake specific actions in their application. Further, flick gestures mayor may not be enabled to provide custom feedback in applications aswell.

Training

FIG. 5 shows an illustrative process for training users to be able toperform a flick gesture. It is appreciated that any number of processesmay be used to train users to perform flick gestures, of which a numberof processes are shown in FIG. 5 with the direct and optional steps.Further, systems may implement flick gestures as described hereinwithout providing training on how to perform flick gestures.

In step 501, the system begins a tutoring process. In step 505, thesystem determines if a user has performed a flick gesture N number oftimes (five or ten or so). If yes, then the system ends the tutorialprocess in step 510. If no from step 505, then the system requests moreflick gestures in order to attempt to train the user. Alternatively, auser may be permitted to opt out from further training in step 509.

FIG. 5 also shows a number of additional optional components that may beincluded in training steps. In step 502, the system may provide an areawhere the user can practice flicks. In step 503, the system may instructa user how to perform a flick gesture. In steps 504 and 508, the systemmay modify the thresholds of FIG. 4 to make flick gesture recognitioneasier or more closely model the actions a user is performing. In step506, the system may request a user to repeat the gesture. Alternatively,or in addition to the request to repeat the gesture in step 506, thesystem may display an indication that the gesture was performedcorrectly. In step 507, the system may repeat instructions to a user onhow to properly perform a flick gesture. The instructions may be generalor may also include an explanation of specific improvements a user canmake based on the previous attempt or attempts at performing a flickgesture. For instance, the process may indicate that the flick gesturewas performed properly or provides them with a helpful tip to perform aflick successfully (e.g., flick faster or flick straighter, etc.).

Illustrative Implementations

FIG. 6 shows illustrative examples of how gesture recognizers may beintegrated with other components. FIG. 6 shows a real time styluscomponent 601 which may attempt to process input from a stylus or otherinput device in real-time.

The real time stylus (RTS) is an object that provides real time stylusevents on a given window handle within a given window input rectangle.The real time stylus can also be considered as a framework to which toadd plug-in objects that handle additional functions. The plug-inobjects may be added and removed as the desired functionality of thereal time stylus changes. The real time stylus may have one or moreinterfaces. In the case of two interfaces, these may be a synchronousinterface and an asynchronous interface. These interfaces providelocations to which plug-ins may be connected to the real time stylus.These interfaces are for illustrative purposes only. Other interfacesmay be used as well.

Plug-ins may be attached to the synchronous and asynchronous interfaces.A plug-in is a functional component that may be added to the real timestylus object. If a plug-in is attached to a synchronous interface ofthe real time stylus object, it may be referred to as a synchronousplug-in. If a plug-in is attached to an asynchronous interface of thereal time stylus object, it may be referred to as an asynchronousplug-in.

The plug-ins may be grouped into collections. A collection of plug-ins,in one or more groups, that are attached to the RTS. Where twocollections exist, they may be associated with the synchronous andasynchronous interfaces of the RTS, respectively. Each collection may beexecuted in order. Where two or more collections exist, they may beexecuted independent of each other because they may be attached todifferent interfaces of the RTS (or RTSs). Data return to the RTS afterbeing handled by each plug-in. The order of plug-ins in the collectionmay affect the output of the collection.

Real time stylus component 601 may include synchronous plug-ins 602-604and asynchronous plug-ins 605-606. The plug-ins may be grouped intoplug-in collections. The synchronous plug-ins 602-604 (including forexample a dynamic renderer) relate to plug-ins that need to quicklyprocess input. Asynchronous plug-ins 605-606 relate to plug-ins on anasynchronous thread of the real time stylus 601. The asynchronousplug-ins 605-606 may process information from the real time stylus on anasynchronous basis.

A flick gesture recognizer may be placed in gesture recognizer 605 andprocessed on the asynchronous thread. Alternatively, the flick gesturerecognizer may be placed on the synchronous thread with othersynchronous plug-ins. When placed on the synchronous thread, the flickgesture recognizer 603 is able to quickly process input from real timestylus 601 prior to that information being processed by additionalplug-ins and sent to the asynchronous thread. Gestures not recognized asflick gestures should be recognized by the gesture recognizer 605 on theasynchronous thread.

The present invention has been described in terms of preferred andexemplary embodiments thereof. Numerous other embodiments, modificationsand variations within the scope and spirit of the appended claims willoccur to persons of ordinary skill in the art from a review of thisdisclosure.

1. A method for training a user how to perform gestures on a userinterface of a computing device, the method comprising: receiving a userinput during a gesture training session, wherein receiving the userinput includes determining when a user input occurs in at least one of agroup of predefined regions of the user interface, wherein eachpredefined region is associated with an action to be fired, eachpredefined region being a section of a circle, the center of the circlebeing a starting point of the user input; mapping at least one of aspeed, a direction or a straightness of the user input in the predefinedregions; determining, by a processor of the computing device, if theuser input is a correctly performed gesture and has been performed afixed number of times, wherein a correctly performed gesture comprisesat least one of a correct speed, a correct direction or a correctstraightness, wherein the user input for the correct speed, the correctdirection, or the correct straightness is received in the group ofpredefined regions, and wherein the user input of the correct speed isdistinguishable from a normal user input, wherein the normal user inputcomprises at least one of a drag-drop, a double actuation, a press andhold, or a hold and drag; providing an instruction to the user on how toproperly perform a flick gesture, wherein the instruction being based onat least one previous attempt by the user to perform the flick gesture,wherein the instruction includes a request to perform the flick gesturefaster, a request to perform the flick gesture straighter and a requestto perform the flick gesture in a specified direction; providing anindication as to whether each gesture has been correctly performed; andending the gesture training session.
 2. The method according to claim 1,further comprising the step of: in response to a negative result fromthe determining step, requesting the user to repeat the input.
 3. Themethod according to claim 1, wherein in response to a negative resultfrom the determining step, indicating that the user input was correctlyrecognized as a gesture.
 4. The method according to claim 1, wherein thegesture is a flick gesture.
 5. The method according to claim 1, furthercomprising the step of: displaying instructions to the user on how tocorrectly perform the gesture.
 6. The method according to claim 1,further comprising the step of: modifying thresholds associated with thedetermining step in response to the user input.
 7. The method accordingto claim 1, further comprising the step of: receiving user input thatindicates the user has opted out of additional training.
 8. The methodaccording to claim 1, wherein receiving the user input includes sensinga user's finger in proximity to a digitizer of the computing device. 9.A system for training a user how to perform gestures, the systemcomprising: an input device for receiving a user input during a gesturetraining session, wherein receiving the user input includes determininga user input occurs in at least one of a group of predefined regions ofthe user interface, wherein each predefined region is associated with anaction to be fired, each predefined region being a section of a circle,the center of the circle being about the starting point of the userinput; a processor in communication with the input device, the processorconfigured to: map at least one of a speed, a direction or astraightness of the user input received at the group of predefinedregions; determine if the user input is a correctly performed gestureand has been performed a fixed number of times, wherein a correctlyperformed gesture comprises at least one of a correct speed, a correctdirection or a correct straightness, wherein the user input for thecorrect speed, the correct direction, or the correct straightness isreceived in the group of predefined regions, and wherein a user input ofthe correct speed is distinguishable from a normal user input whereinthe normal user input comprises at least one of a drag-drop, a doubleactuation, a press and hold, or a hold and drag; provide an instructionto the user on how to properly perform a flick gesture, wherein theinstruction being based on at least one previous attempt by the user toperform the flick gesture, wherein the instruction includes a request toperform the flick gesture faster and a request to perform the flickgesture straighter than at least one previous attempt by the user;provide an indication as to whether each gesture has been correctlyperformed; and end the gesture training session.
 10. The systemaccording to claim 9, wherein in response to the user inputcorresponding to a negative result, the processor is configured torequest the user to repeat the input in response to determining.
 11. Thesystem according to claim 9, wherein in response to the user inputcorresponding to a negative result, the processor is configured toindicate to the user that the user input was correctly recognized as agesture.
 12. The system according to claim 9, wherein the gesture is aflick gesture.
 13. The system according to claim 9, further comprising,a display device that displays instructions to the user on how tocorrectly perform the gesture.
 14. The system according to claim 9,wherein the processor is further configured to modify thresholdsassociated with the determination if the user input is a correctlyperformed gesture.
 15. The system according to claim 9, wherein theinput device is further configured to receive user input that indicatesthe user has opted out of additional training.
 16. The system accordingto claim 9, wherein the user input is received via a user's finger inproximity to the input device.
 17. A computer-readable storage mediumhaving a program stored thereon, the program for training a user how toperform gestures, the program configured to be executed by a processorof a computing device, the method comprising: receiving a user inputduring a gesture training session, wherein receiving the user inputincludes determining when a user input occurs in at least one of a groupof predefined regions, wherein each predefined region is associated withan action to be fired, each predefined region being a section of acircle, the center of the circle being about the starting point of theuser input; mapping at least one of a speed, a direction or astraightness of the user input received at the group of predefinedregions; determining if the user input is a correctly performed gestureand has been performed a fixed number of times, wherein a correctlyperformed gesture comprises at least one of a correct speed, a correctdirection or a correct straightness, wherein the user input for thecorrect speed, the correct direction, or the correct straightness isreceived in the group of predefined regions and wherein the user inputbeing of the correct speed is distinguishable from a normal user input,wherein the normal user input comprises at least one of a drag-drop, adouble actuation, a press and hold, or a hold and drag; providing aninstruction to the user on how to properly perform a flick gesture,wherein the instruction being based on at least one previous attempt bythe user to perform the flick gesture, wherein the instruction includesa request to perform the flick gesture faster and a request to performthe flick gesture in a specified direction; providing an indication asto whether each gesture has been correctly performed; and ending thegesture training session.
 18. The computer-readable storage mediumaccording to claim 17, further comprising, in response to a negativeresult from the determining step, requesting the user to repeat theinput.
 19. The computer-readable storage medium according to claim 17,wherein in response to a negative result from determining if the userinput is a correctly performed gesture and has been performed a fixednumber of times, the indication indicates to the user that the userinput was correctly recognized as a gesture.
 20. The computer-readablestorage medium according to claim 17, wherein the gesture is a flickgesture.
 21. The computer-readable storage medium according to claim 17,further comprising displaying instructions to the user on how tocorrectly perform the gesture.
 22. The computer-readable storage mediumaccording to claim 17, further comprising modifying thresholdsassociated with the determining step in response to the user input. 23.The computer-readable storage medium according to claim 17, furthercomprising receiving user input that indicates the user has opted out ofadditional training.
 24. The computer-readable storage medium accordingto claim 17, wherein receiving the user input includes sensing a user'sfinger by the computing device.
 25. A system for training a user toperform a gesture, the system comprising: an input that receiveselectronic ink created by a user during a gesture training session,wherein receiving the user input includes determining when a user inputoccurs in at least one of a group of predefined regions of the userinterface, the electronic ink pertaining to a training session, whereineach predefined region is associated with an action to be fired, eachpredefined region being a section of a circle, the center of the circlebeing about a starting point of the user input; a storage; and aprocessor connected to the input and the storage, the processorconfigured to operate in one of an operational mode and a training mode,where in the training mode, the processor is configured to: map at leastone of a speed, a direction, or a straightness of the electronic ink, atthe group of predefined regions; determine if the electronic ink is aproperly performed gesture, wherein a properly performed gesturecomprises at least one of a correct speed, a correct direction or acorrect straightness, wherein the user input for the correct speed, thecorrect direction, or the correct straightness is received in the groupof predefined regions, and wherein a user input being of the correctspeed is distinguishable from a normal user input wherein the normaluser input comprises at least one of a drag-drop, a double actuation, apress and hold, or a hold and drag; provide an instruction to the useron how to properly perform a flick gesture, wherein the instructionbeing based on at least one previous attempt by the user to perform theflick gesture, wherein the instruction includes a request to perform theflick gesture straighter than at least one previous attempt by the userand a request to perform the flick gesture in a specified direction;provide an indication as to whether the gesture has been correctlyperformed; and determine that the gesture has been performed a fixednumber of times.
 26. The system according to claim 25, wherein theproperly performed gesture is a flick gesture.
 27. The system accordingto claimed 25, further comprising: a display; and a digitizer associatedwith the display, the digitizer creating the electronic ink by sensingmovement of a stylus controlled by the user.
 28. The system according toclaimed 25, further comprising: a display; and a digitizer associatedwith the display, the digitizer creating the electronic ink by sensingmovement of a user's finger.